Object Localization by Joint Audio-Video Signal Processing

There are many di erent approaches either for the localization of sound sources or for tracking of visible objects in image sequences. However, most applications use only one modality, that is, they process only audio or video information for object localization. In this paper we introduce a method for the estimation of object positions based on joint audio-video information. The key technique is a modi ed decentralized Kalman lter (MDKF), where the object localization problem is viewed as state estimation. First, the position of the object is estimated based on audio and video information separately. Then, the locally estimated results are further processed in a decentralized Kalman lter for data fusion. At the output we obtain the joint estimation results. Experiments have shown that the joint estimation provides more correct localization results than obtained by using audio or video information only.